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Results  in  Brief 


United  States 

General  Accounting  Office 

Washington,  D.C.  20548 


Accounting  and  Information 
Management  Division 

B-259196 

December  21, 1994 

The  Honorable  Edward  J.  Markey 
Chairman 

The  Honorable  Jack  Fields 
Ranking  Minority  Member 
Subcommittee  on  Telecommunications 
and  Finance 

Committee  on  Energy  and  Commerce 
House  of  Representatives 

This  report  responds  to  your  August  2, 1994,  letter  requesting  that  we 
review  recent  outages  experienced  by  the  National  Association  of 
Securities  Dealers  (nasd)  automated  quotation  and  trading 
systems — commonly  called  nasdaq.  Specifically,  in  your  letter  and  in 
subsequent  meetings  with  your  office,  you  asked  us  to  determine  (1)  the 
nature  and  causes  of  the  outages  of  July  14  and  15,  and  August  1, 1994, 

(2)  the  impact  of  the  outages  on  market  participants,  (3)  the  adequacy  of 
nasd’s  approach  to  respond  to  contingencies  and  disasters,  (4)  how  well 
NASD  oversees  its  automated  systems  and  facihties,  and  (5)  how  well  the 
Securities  and  Exchange  Commission  (sec)  is  ensuring  that  the  securities 
markets  are  prepared  for  contingencies  and  disasters. 


The  NASDAQ  system  outages  on  July  14  and  15,  and  August  1, 1994,  were 
caused  by  unrelated  software  and  hardware  malfunctions.  These  outages 
had  limited  impact  on  individual  investors  and  derivatives  markets  but 
hampered  the  abUity  of  broker-dealers  to  perform  best  and  efficient  trade 
executions.  While  nasd  takes  the  rehabdity  of  its  systems  very  seriously, 
these  recent  outages  and  associated  malfunctions  point  to  areas,  such  as 
testing,  where  further  improvement  is  needed  to  guard  against  the  risk  of 
recurrence.  In  addition,  while  nasd  has  a  separate,  backup  computer 
facility  in  case  of  contingencies,  control  weaknesses  at  this  facility  and  in 
nasd’s  contingency  and  disaster  plan  could  make  it  difficult  for  nasd  to 
recover  quickly  when  exigencies  occur.  Finally,  nasd’s  oversight  of 
systems  is  hmited  by  the  fact  that  its  internal  audit  function  generally  does 
not  include  the  review  of  market  systems  in  the  scope  of  its  work. 

Compounding  these  problems  is  the  fact  that  while  sec  has  strengthened 
oversight  of  market  automation  in  such  areas  as  contingency  planning, 
gaps  exist  in  its  oversight  program.  For  example,  sec  does  not  always 
follow  up  to  ensure  auditors’  recommendations  are  carried  out.  Until  sec 
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fills  these  gaps,  it  caimot  ensure  that  it  is  adequately  overseeing  the  rapid 
growth  of  automation  in  the  securities  industry. 


Background 


Established  in  1939,  nasd  regulates  (1)  over-the-counter  securities  trading 
(that  is,  trading  that  does  not  occur  on  the  floor  of  a  stock  exchange)  and 
(2)  all  brokers  and  dealers  conducting  securities  business  with  the  pubhc. 
NASD  owns  and  operates  NASDAQ,  a  computerized  communication  system 
that  provides  quotation  information  on  and  facilitates  trade  executions  for 
5,700  securities.  Implemented  in  1971,  NASDAQ  links  a  nationwide  network 
of  about  500  brokerage  firms,  called  market  makers.  These  firms  maintain 
inventories  of  securities  which  they  buy  from  or  sell  to  investors. 

During  1993,  66.5  bdlion  shares  of  stock — totaling  $1.35  trillion — were 
traded  in  this  market.  These  volumes  represent  43.6  percent  of  the  total 
shares  traded  on  U.S.  stock  markets,  or  about  32.6  percent  of  the  total 
doUar  value  traded. 

nasd’s  headquarters  is  located  in  Washington,  D.C.  The  Association’s 
automated  quotation  and  trading  systems  are  located  in  and  operated  firom 
its  primary  data  processing  facUity  in  Trumbull,  Connecticut.  Its  backup 
systems  are  located  at  nasd’s  data  processing  facility  in  Rockville, 
Maryland,  which  also  houses  automated  administrative  systems  such  as 
payroll,  personnel,  and  market  surveillance. 

The  U.S.  securities  markets  are  primarily  governed  by  self-regulatory 
organizations,  such  as  nasd,  which,  in  tiun,  are  overseen  by  sec.  While 
self-regulatory  organizations  are  responsible  for  maintaining  smooth  and 
dependable  operations  with  their  automated  systems,  sec  is  responsible 
for  overseeing  overall  market  operations,  including  systems  used  to 
support  such  operations. 


Scope  and 
Methodology 


To  determine  the  nature  and  causes  of  the  outages  and  to  better  learn  how 
NASD  develops,  tests,  and  operates  systems,  we  interviewed  nasd  senior 
officials,  including  the  Executive  Vice  President  and  Chief  Technology 
Officer,  the  Senior  Vice  President  for  Production  Services,  the  Vice 
President  for  Computer  Operations,  the  Director  for  Quality  Assurance, 
and  the  Director  for  Performance  Measurement.  In  addition,  these  officials 
provided  us  with  a  minute-by-minute  chronology  of  events  as  they 
occurred  on  July  14  and  15  and  August  1.  We  also  obtained  and  reviewed 
nasd’s  pohcies  and  procedures  for  quality  assurance  and  stress  testing. 
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Finally,  we  visited  the  primary  data  processing  facility  in  Trumbull, 
Connecticut,  to  observe  the  systems  and  the  controls  used  to  safeguard 
them. 

We  determined  the  impact  of  the  recent  NASD  system  outages  on  three 
categories  of  market  participants — ^market  makers,  derivatives  markets, 
and  individual  investors.  For  the  market  makers,  we  used  a  structured 
questionnaire  to  collect  information  from  the  top  12  market 
makers — Merrill  Lynch;  Smith  Barney  Shearson;  Herzog,  Heine,  Geduld; 
Mayer  &  Schweitzer;  Troster  Singer  Corporation;  Goldman,  Sachs  & 
Company;  Lehman  Brothers;  Morgan  Stanley  &  Company;  Bear,  Steams  & 
Company;  The  First  Boston  Corporation;  PaineWebber;  and  Sherwood 
Securities  Corporation.  Together,  these  12  represent  over  60  percent  of 
nasd’s  total  trading  volume. 

Our  questionnaire  included  inquiries  on  how  the  outages  impacted  the 
market  makers’  ability  to  obtain  information  and  execute  trades,  as  well  as 
questions  on  the  impact  of  the  outages  on  confidence  in  nasd  systems  and 
the  market,  and  on  future  participation  in  this  stock  market.  We  met  with 
six  of  the  market  makers  and  mailed  the  questionnaire  to  the  others. 

For  the  derivatives  markets,  we  interviewed  nasd  officials  including  the 
Chief  Operating  Officer,  Chief  Technology  Officer,  and  Chief  Economist, 
as  weU  as  senior  officials  from  the  Chicago  Board  Options  Exchange  and 
the  Chicago  Mercantile  Exchange. 

Finally,  to  assess  the  impact  of  the  outages  on  individual  investors,  we 
interviewed  market  makers,  senior  nasd  officials,  and  officials  from  the 
National  Association  of  Investors  Corporation  and  the  American 
Association  of  Individual  Investors — ^representing  about  440,000  members 
combined. 

In  assessing  the  adequacy  of  nasd’s  plans  to  respond  to  contingencies  and 
disasters,  we  conducted  a  walk-through  of  nasd’s  backup  facility  in 
Rockville,  Maryland.  We  also  interviewed  those  nasd  officials  responsible 
for  preparing,  maintaining,  and  testing  the  Association’s  contingency  and 
disaster  recovery  plan.  In  addition,  we  reviewed  nasd’s  contingency  and 
disaster  recovery  plan  and  processes,  including  examinations  of  the 
1993-94  test  schedules  and  results. 

To  determine  how  well  nasd  oversees  its  automated  market  systems  and 
facilities,  we  examined  the  role  of  nasd’s  Internal  Review  office  and 
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System  Outages  Due 
to  Software  and 
Hardware 
Malfunctions 


discussed  the  work  it  has  done  in  the  past  and  is  planning  to  undertake  in 
the  future.  We  interviewed  the  Director  of  Internal  Review  and  reviewed 
position  descriptions  for  the  auditors  who  review  nasd’s  systems  and 
computer  facilities.  We  also  obtained  and  examined  Internal  Review’s 
audit  plan  detaUing  the  scope  of  work  to  be  performed  through  April  1995. 
Finally,  we  interviewed  systems  managers  at  both  the  primary  and  backup 
computing  facilities  to  determine  the  extent  of  their  involvement  with 
Internal  Review. 

To  determine  how  sec  generally  oversees  markets’  preparedness  for 
contingencies  and  disasters,  we  interviewed  senior  officials  in  sec’s 
Division  of  Market  Regulation  and  obtained  a  chronology  of  events  and 
supporting  documentation  regarding  the  Commission’s  role  and  response 
to  the  NASD  outages  experienced  during  July  14  and  15  and  August  1.  In 
addition,  we  reviewed  sec’s  automation  review  policy,  the  Commission’s 
report  of  its  most  recent  inspection  at  nasd  which  occurred  in  1992,  and 
the  audit  report  of  the  most  recent  review  of  automated  NASD  systems 
conducted  by  an  independent  public  accountant  in  1992. 

We  conducted  our  review  from  August  through  October  1994,  in 
accordance  with  generally  accepted  government  auditing  standards.  We 
discussed  the  contents  of  this  report  with  senior  officials  firom  nasd  and 
sec’s  Division  of  Market  Regulation  and  incorporated  their  comments 
where  appropriate. 


The  system  outages  experienced  by  nasd  in  July  and  August  were  due  to 
malfunctioning  software  and  hardware.  Specifically,  on  July  14,  new 
communications  software  being  implemented  as  part  of  nasd’s  efforts  to 
upgrade  its  system  did  not  operate  as  intended  and  caused  the  system  to 
fad.  When  nasd  staff  restarted  the  system,  the  communications  software 
experienced  additional  problems. 

Consequently,  nasd  shut  down  the  system  and  reconfigured  it  to  use  the 
old  communications  software,  nasd  operated  its  system  this  way  for  the 
remainder  of  the  trading  day  with  only  minor  problems.  In  total,  the 
outages  caused  the  system  to  be  down  for  about  14  minutes.  According  to 
NASD  systems  officials,  they  corrected  one  problem  with  the  new 
communications  software  that  evening.  These  officials  also  told  us  that 
they  disabled  a  function  of  the  new  communications  software  that  was 
causing  a  second  problem  that  could  not  be  fixed  immediately,  and 
reconfigured  the  system  to  use  the  old  software  for  this  function. 
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On  July  16,  before  the  normal  market  opening  (9:30  am.  EST),  the 
system’s  response  time  slowed  to  unacceptable  levels  while  processing 
routine  tasks,  prompting  NASD  to  delay  opening  the  market.  At  about  11:00 
am.,  NASD  diagnosed  the  problem  as  a  faulty  hardware  component  (used  to 
manage  disk  access  and  storage  devices),  took  it  off-line,  and  opened  the 
market  at  11:65  am.,  approximately  2-1/2  hours  late.  According  to  NASD 
systems  officials,  it  took  them  about  1-1/2  hours  after  normal  opening  time 
to  locate  this  problem  because  their  focus  was  on  the  new  software,  while 
the  problem  was  actually  caused  by  an  intermittent  hardware  failure. 

After  the  market  opened,  nasd  then  opened  a  market  function  for 
exchange-listed  securities — the  Consohdated  Quotation  Service.  Because 
this  function  had  been  closed  for  the  morning,  the  transaction  rate  surged. 
Software  controls  in  the  communications  software  that  were  designed  to 
manage  (limit)  the  number  of  transactions  the  system  would  accept  did 
not  fully  protect  the  system  from  this  surge  and  transaction  backlogs 
began  to  build,  resulting  in  the  system’s  response  time  increasing,  nasd 
responded  by  turning  off  selected  automated  services  to  reduce  the 
processing  workload.  The  system  functioned  with  only  minor  problems  for 
the  rest  of  the  trading  day. 

The  outage  on  August  1,  which  lasted  34  minutes,  was  caused  by  a  faulty 
circuit  board  in  nasd’s  backup  electrical  system.  The  backup  electrical 
system,  which  consists  of  commercial-grade  batteries  and  generators 
owned  and  operated  by  nasd,  was  activated  because  the  power  from  the 
local  utility  company  dipped  to  an  unacceptable  level.  As  designed,  the 
backup  battery  system  operated  until  the  facility  could  be  switched  over  to 
the  backup  generators;  however,  during  the  switchover,  the  circuit  board 
responsible  for  monitoring  the  conversion  malfunctioned.  This  resulted  in 
a  total  loss  of  power  to  the  data  center.  At  this  point,  nasd  switched 
operations  to  the  backup  data  processing  facihty  in  Maryland  and 
continued  operations  for  the  rest  of  the  trading  day. 

To  address  this  problem,  nasd  (1)  replaced  the  circuit  board,  (2)  had  the 
contractor  who  supplied  the  backup  electrical  system  determine  why  the 
board  malfunctioned,  (3)  is  considering  purchasing  a  second  backup 
electrical  system  of  batteries,  generators,  and  circuit  board  to  supplement 
its  existing  backup  electrical  system,  and  (4)  hired  a  contractor  to  assess 
other  single  points  of  failure  in  the  backup  electrical  system. 
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Incomplete  Testing 
May  Have  Prevented 
NASD  From  Detecting 
Software 
Malfunctions 


The  malfunctions  that  caused  the  July  outages  might  have  been  avoided 
had  NASD  more  thoroughly  tested  its  software.  Testing  systems  to  assess 
their  ability  to  operate  as  intended  and  process  unusually  large 
workloads — commonly  referred  to  as  quality  assurance  and  stress  testing, 
respectively — helps  identify  and  correct  system  weaknesses  before  they 
cause  data  processing  disruptions  in  a  live  operating  environment. 

NASD  performs  quality  assurance  and  stress  tests  on  its  systems.  For 
instance,  quality  assurance  personnel  test  software  to  determine  whether 
it  meets  estabhshed  business  requirements.  However,  nasd’s  quality 
assurance  testing  was  limited  in  scope.  Specifically,  quality  assurance  did 
not  (1)  test  aU  requirements  and  (2)  verify  that  the  system  would  not 
operate  in  m£q)propriate  ways.  For  example,  one  requirement  of  nasd’s 
communications  software  was  to  hmit  the  total  number  of  transactions  the 
system  could  accept  into  its  processing  queue;  however,  quality  assurance 
did  not  test  this  softw2ire  function.  On  July  15  when  nasd  opened  the 
market,  the  system  accepted  more  transactions  than  it  was  designed  to 
handle  without  having  to  re-queue  transactions,  which  slowed  system 
processing  speed  to  unacceptable  levels. 

In  addition,  nasd  has  a  Performance  Measurement  Unit  responsible  for 
stress  testing.  This  unit  tests  systems  to  determine  how  they  behave  under 
high  workloads  and  demanding  conditions.  However,  these  tests  were  also 
limited  in  scope.  For  example,  nasd  did  not  test  the  system  with  sufficient 
volume  to  drive  the  system  beyond  the  point  where  it  begins  to  re-queue 
transactions,  nor  with  a  heavy  backlog  of  transactions,  such  as  occurred 
on  July  15. 

NASD  systems  officials  said  that  their  quality  assurance  testing  program  is 
rigorous  enough  to  catch  most  problems,  but  acknowledged  that  problems 
sometimes  can  go  undetected.  In  addition,  while  these  officials  said  that 
their  stress  testing  is  adequate,  they  also  agreed  that  their  stress  tests 
could  be  expanded  to  include  transaction  backlog  conditions  similar  to 
those  experienced  on  July  15. 

We  also  discussed  with  systems  officials  why  they  installed  new 
communications  software  on  Friday,  July  15,  a  “double  witching”  day.  On 
such  Fridays,  the  market  is  potentially  volatile  because  options  and  other 
related  financial  instruments  expire  and  market  participants  may  need  to 
buy  or  sell  stocks  to  meet  obhgations.  Installing  new  software  on  such 
potentially  volatile  days  increases  the  risk  that  system  problems  could 
worsen  market  conditions  and  therefore,  should  be  avoided,  sec  has  also 
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found  this  practice  to  be  undesirable  and  has  recommended  since  the  July 
and  August  outages  that  nasd  avoid  installing  new  software  on  such  days 
and  that  systems  managers  coordinate  changes  with  top  management  at 
NASD  headquarters  who  are  knowledgeable  about  market  conditions. 

NASD  systems  officials  told  us  that  it  is  nasd’s  pohcy  not  to  install  system 
changes  on  Fridays.  While  the  installation  of  the  new  communications 
software  carried  over  to  July  15,  due  to  the  problems  on  July  14,  these 
officials  believed  that  there  was  minimal  risk  of  a  malfunction  because 
(1)  the  software  had  been  tested,  (2)  it  was  installed  in  a  phased  approach 
over  a  2-week  period,  and  (3)  they  were  confident  in  the  systems  and  the 
personnel  who  developed  and  operate  the  software.  Additionally,  the 
systems  officials  told  us  that  they  do  discuss  system  changes  with 
business  managers  at  the  primary  site. 

During  the  course  of  our  work,  nasd  systems  managers  began  to  inform 
top  business  managers  via  electronic  mail  of  all  upcoming  system  changes 
and  installation  schedules.  This  notwithstanding,  unless  NASD  strictly 
adheres  to  its  policy  of  avoiding  the  installation  of  new  software  changes 
on  potentially  volatile  days,  it  risks  having  system  malfunctions  exacerbate 
market  conditions.  This  risk  could  be  made  greater  by  the  fact  that  nasd 
will  be  making  numerous  changes  as  it  upgrades  its  system. 


Impact  of  Outages  on 
Market  Participants 
and  Related  Markets 
Varied 


The  recent  outages  experienced  by  nasd  had  varying  effects  on  market 
participants  and  derivatives  markets.  For  example,  individual  investors 
were  not  significantly  affected  and  did  not  report  complaints  regarding  the 
outages.  Conversely,  market  makers  were  impacted  because  they  did  not 
have  the  benefit  of  nasd’s  automated  quotation  and  trading  system  to 
conduct  business.  In  general,  the  impact  was  greatest  on  July  15  when  the 
system  was  out  for  2-1/2  hours.  Nonetheless,  213  million 
shares — 79  percent  of  the  average  daily  volume  for  July— were  traded  that 
day. 


Market  Makers  Unable  to 
Perform  Best  and  Efficient 
Trade  Executions 


Market  makers  surveyed  characterized  the  impact  of  the  outages  as  being 
veiy  great  because  they  could  not  obtain  updated  price  quotations  firom 
NASD.  Without  this  information,  market  makers  were  unable  to  facilitate 
the  best  and  efficient  execution  of  trades.  Market  makers  stated  that  the 
July  15  outage  was  particularly  severe  because  they  could  not  get  quote 
information  for  the  first  2-1/2  hours.  Their  frustration  was  heightened  by 
the  fact  that  they  were  uncertain  when  the  market  would  reopen. 
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However,  to  serve  customers  who  were  willing  to  buy  or  sell  without 
updated  quotes,  market  makers  relied  on  other  means,  such  as  using 
broker-dealer  owned  trading  systems  (Instinct,^  for  example)  to  execute 
trades.  Market  makers  generally  told  us  that  the  2-1/2-hour  delay  of  July  15 
resulted  in  lost  opportunities  to  do  business.  In  addition,  seven  market 
makers  stated  that  they  lost  revenue  they  collect  for  executing  trades  on  a 
normal  day  due  to  the  outage.  Of  these  seven,  three  estimated  that  they 
lost  20  to  26  percent  of  such  fees,  while  the  remaining  four  firms  had  not 
estimated  the  extent  of  their  monetary  losses  attributable  to  the  outage. 


Effect  on  Individual  According  to  securities  industry  officials  we  interviewed,  the  outages  had 

Investors  Limited  impact  on  individual  investors,  who  hold  about  55  percent  of  all  NASD 

market  stocks.  First,  officials  from  two  nonprofit  associations, 
representing  about  440,000  individual  investors,  told  us  that  while  their 
members  generally  report  events  that  affect  them,  no  complaints  were 
reported  regarding  the  NASD  outages.  One  official  stated  that  individual 
investors  tend  to  make  long-term  investments  so  that  outages  of  1  day 
would  probably  not  affect  them. 

Second,  according  to  nasd,  the  mcyority  of  individual  investors  who 
participate  in  the  stock  markets  do  so  through  mutual  funds,  which 
generally  price  their  funds  using  end-of-day  stock  quotes.  Since  nasd 
provided  end-of-day  stock  quotes  on  the  days  the  system  experienced 
outages,  mutual  funds,  and  thus  most  individual  investors,  were 
unaffected.  Finally,  one  of  the  market  makers  whose  business  caters  to 
individual  investors  told  us  that  all  of  its  trades  were  executed,  although 
not  immediately,  given  the  unavailability  of  updated  quotes  to  guarantee 
best  price  execution  of  trades. 


Derivatives  Market  Trading  The  derivatives  markets — such  as  the  options  and  futures  markets — trade 
Halted  Without  Quotes  products  that  derive  their  value  from  nasd  and  other  markets’  stocks.  The 

Chicago  Board  Options  Exchange,  which  trades  the  largest  number  of 
NASD  stock  options,  had  to  stop  trading  these  instruments  on  July  16  and 
August  1  because  quotation  information,  which  is  used  to  derive  the  price 
of  options,  was  not  available.  In  addition,  according  to  options  exchange 
officials,  when  nasd  opened  its  market  2-1/2  hours  later  on  July  15  and 
began  transmitting  quote  information,  the  exchange  encountered  a  large 
volume  of  orders  that  had  to  be  processed  in  a  relatively  short  time  frame. 


^Instinet  is  a  network  of  computer  terminals  that  facilitates  the  trading  process  by  matching  buyers 
with  sellers.  Instinet  is  registered  with  the  SEC  as  a  broker-dealer. 
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We  also  interviewed  officials  at  the  Chicago  Mercantile  Exchange,  which 
trades  the  largest  number  of  futures  on  stock  indices  whose  values  are 
derived  from  the  value  of  stocks  traded  in  nasd’s  market.  These  officials 
stated  that  the  outages  had  no  discernable  impact  on  their  trading 
operations  because  the  vendors  who  price  the  indices  continued  to  do  so 
using  last  available  quote  information  from  nasd. 


While  NASD  Prepares 
for  Contingencies, 
Management  and 
System  Control 
Weaknesses  Exist 


NASD  has  taken  significant  steps  to  prepare  its  systems  for  contingencies 
and  disasters.  It  operates  a  backup  computer  facihty  to  be  used  if  there  are 
problems  or  outages  at  the  primary  computer  site.  In  addition,  nasd  has 
prepared  a  detailed  plan  that  identifies  critical  operations  and  the  key 
individuals  responsible  for  carrying  out  specified  procedures  during 
emergencies,  such  as  power  outages  and  natural  disasters.  The 
Association  also  conducts  tests  to  gauge  staff  preparedness. 


However,  there  are  management  and  control  weaknesses  in  nasd’s 
contingency  and  disaster  recovery  activities.  While  these  weaknesses  did 
not  contribute  to  the  problems  experienced  on  July  14  and  15  and 
August  1,  they  make  nasd  vulnerable  to  problems  should  emergencies 
occur.  For  example,  the  contingency  and  disaster  recovery  plan  is 
incomplete  and  out  of  date.  Certain  contingency  scenarios  have  not  yet 
been  drafted  and  incorporated  into  the  plan  and  names  of  some 
emergency  personnel,  who  are  no  longer  in  such  positions,  have  not  been 
updated.  In  addition,  the  plan  does  not  clearly  delineate  who  is  responsible 
for  making  systems  decisions  during  contingencies  and  disasters.  During 
our  limited  scope  review  at  the  backup  site,  we  also  identified  certain 
internal  control  weaknesses.  For  example,  the  data  center  is  located  over 
a  storage  room  of  paper  products,  posing  a  potential  fire  hazard. 


During  the  course  of  our  work,  we  brought  these  management  and  control 
weaknesses  to  nasd’s  attention  and  they  attributed  the  weaknesses  to 
oversights  on  their  part,  nasd  agreed  to  correct  them  immediately. 


NASD  Oversight  of 
Systems  Is  Limited 


We  examined  the  role  of  nasd’s  internal  audit  function  in  identifying 
system  and  control  weaknesses.  We  found  that  despite  nasd’s  extensive 
reliance  on  automated  systems  to  accomplish  its  mission,  until  recently  it 
had  only  one  auditor  with  computer  expertise  reviewing  automated 
systems.  In  addition,  the  scope  of  the  auditor’s  work  was  generally  limited 
to  reviewing  administrative  systems  located  at  the  backup  site. 
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According  to  NASD  officials,  while  internal  audit’s  focus  has  been  on 
administrative  systems,  it  has  performed  some  work  on  market-related 
systems.  Specifically,  internal  audit  was  involved  during  the  development 
and  implementation  of  the  Fixed  Income  Pricing  System  and  has  reviewed 
Small  Order  Execution  System  outj^es.  In  addition,  the  internal  audit 
function  was  established  2  years  ago  and  is  stiU  in  the  process  of 
establishing  a  program  to  ensure  adequate  audit  coverg^e.  Finally,  internal 
audit  focused  its  work  on  administrative  systems  at  the  backup  site 
because  these  systems  were  judged  to  be  more  at  risk  than  the  market 
systems.  This  decision  was  based  in  part  on  the  fact  that  the  market 
systems  had  been  reviewed  by  an  external  auditor  in  1992  and  internal 
audit  believed  it  could  rely  on  this  work. 

Because  regular  external  and  internal  reviews  are  complementary 
management  control  practices  used  to  oversee  the  use  of  automated 
systems,  reviews  by  an  external  auditor  are  not  a  complete  substitute  for 
the  day-to-day  audit  coverage  provided  by  internal  audit.  Recognizing  this, 
NASD  officials  stated  that  they  (1)  recently  hired  a  second  internal  auditor 
with  computer  expertise,  (2)  plan  to  expand  coverage  of  market  systems 
in  the  audit  work  plan  for  the  upcoming  year,  and  (3)  are  discussing  with 
SEC  the  frequency  of  external  reviews. 


SEC  Has  Issued 
Guidance  on 
Contingency 
Preparation  but 
Oversight  Gaps 
Remain 


Our  past  reviews  of  automated  stock  market  systems  have  identified  the 
need  for  sec  to  establish  the  capability  to  address  such  technical  issues  as 
contingency  and  disaster  recovery  planning.^  sec  has  subsequently  taken 
steps  to  improve  its  oversight  of  the  markets’  use  of  automation.  For 
example,  the  Commission  established  an  Office  of  Automation  and 
International  Markets  and  issued  an  automation  review  policy  that 
encourages  the  securities  markets  to  perform  independent  reviews  of  their 
automated  systems  and  operations  in  such  areas  as  contingency  and 
disaster  planning.^  The  Commission  planned  to  measure  compliance  with 
the  policy  by  conducting  inspections  on  a  periodic  basis. 

However,  gaps  exist  in  sec’s  oversight  program.  First,  it  is  unclear  how 
often  SEC  expects  the  markets  to  perform  independent  automation  reviews 
because  the  Commission’s  policy  does  not  state  a  specific  frequency 


^Financial  Markets:  Computer  Security  Controls  at  Five  Stock  Exchanges  Need  Strengthening 
(GAO/IMTEC-91-56,  August  28,  1991)  and  Financial  Markets:  Active  Oversight  of  Market  Automation 
by  SEC  and  CFTC  Needed  (GAO/IMTEC-91-21,  April  2,  1991). 

^Securities  and  Exchange  Commission  Release  No.  34-27445,  54  Fed.  Reg.  48703  (1989),  and  No. 
34-29185,  56  Fed.  Reg.  22490  (1991). 
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requirement  For  example,  the  last  such  review  at  nasd  was  performed  in 
1992.  In  addition,  sec  has  not  established  how  frequently  it  will  perform  its 
inspections.  In  practice,  its  inspections  have  been  limited  to  about  every  3 
years  because  the  Commission  only  has  four  technical  staff  members 
capable  of  conducting  this  work.  Further,  the  scope  of  sec’s  inspections 
has  been  limited  to  reviewing  other  auditors’  work,  rather  than  conducting 
first-hand  reviews  of  system  safeguards. 

SEC  also  has  not  always  followed  up  to  ensure  auditors’  recommendations 
are  carried  out.  For  example,  in  1992,  nasd  had  an  external  auditor  review 
its  systems.  This  auditor  identified  a  serious  control  weakness  in  the  way 
NASD  modified  software  on  the  production  system  during  emergencies.  As 
part  of  its  1992  inspection,  sec  reviewed  the  audit  report,  agreed  with  the 
auditor’s  finding,  and  recommended  that  nasd  take  countermeasures  to 
mitigate  this  weakness.  However,  at  the  time  of  our  work,  nasd  had  not  yet 
corrected  this  weakness.  While  sec  officials  told  us  that  it  is  their  goal  to 
discuss  all  unresolved  audit  findings  with  nasd  and  the  other  markets 
during  periodic  briefings  on  market  automation  developments,  these 
officials  acknowledged  that  they  had  not  taken  action  to  ensure  nasd  had 
implemented  the  recommendation. 

Officials  in  sec’s  Market  Regulation  Division  stated  that  the  automation 
review  policy  is  still  evolving  and  for  this  reason,  they  have  not  yet 
finalized  aU  of  its  requirements.  For  example,  sec  staff  are  currently 
negotiating  with  the  securities  markets  to  determine  how  often  the 
external  reviews  wUl  be  performed  and  expect  to  reach  agreement  soon. 

In  addition,  the  Market  Regulation  officials  advised  us  that  they  are  now 
including  first-hand  reviews  of  selected  system  safeguards  as  part  of  their 
inspections. 

However,  these  officials  told  us  that  they  are  unclear  what  the  optimal 
frequency  for  inspections  should  be.  They  also  told  us  that  sec  would  be 
unable  to  conduct  more  frequent  inspections  and  other  oversight  activities 
because  it  only  has  four  computer  specialists  to  oversee  market 
automation  at  over  19  markets  and  other  related  organizations,  such  as 
clearing  agencies  and  depositories.  In  addition,  they  said  that  hiring 
additional  staff  has  been  deferred  by  the  Division  due  to  other  priorities. 
On  November  17, 1994,  these  officials  told  us  that  they  had  recently 
received  authority  to  hire  two  additional  technical  staff  and  were  in  the 
process  of  advertising  the  positions.  UntQ  sec  determines  the  appropriate 
frequency  of  inspections,  it  cannot  be  sure  it  has  the  correct  number  of 
technical  staff  to  oversee  automated  market  systems. 
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Conclusions 


NASD  is  aware  of  the  importance  of  maintaining  reliable  systems  and 
providing  backup  in  the  case  of  emergencies,  and  is  taking  action  to 
correct  the  weaknesses  identified  in  this  report.  Addressing  these 
weaknesses  will  lower  the  risk  of  future  outages  and  enable  nasd  to 
respond  more  quickly  and  appropriately  to  future  contingencies  and 
disasters. 

While  SEC  has  made  progress  in  strengthening  oversight  of  market 
automation,  gaps  still  exist  in  its  oversight  program.  Until  sec  fills  these 
gaps,  the  Commission  cannot  ensure  that  it  is  adequately  overseeing  the 
rapid  growth  of  automation  in  the  securities  industry. 


Recommendations 


We  recommend  that  the  Chairman,  sec,  ensure  that  nasd 

•  expands  testing  processes  for  its  market  systems  to  better  detect 
problems; 

•  performs  a  thorough  assessment  of  its  existing  systems  environment  to 
identify  weaknesses; 

•  avoids  implementing  software  changes  on  potentially  volatile  trading  days; 

•  corrects  weaknesses  in  its  contingency  and  disaster  recovery  plan  and 
backup  data  processing  facility;  and 

•  regularly  schedules  and  conducts  audits  of  its  market  systems. 

In  addition,  we  recommend  that  sec’s  Chairman  (1)  reach  agreement  with 
securities  markets  on  the  frequency  of  independent  reviews,  (2)  determine 
SEC  inspection  frequency  needed  to  ensure  adequate  oversight  of  market 
systems  and  facilities,  and  (3)  foUow  up  on  systems  auditors’ 
recommendations  and  ensure  that  the  recommendations  are  adequately 
resolved.  Given  that  the  gaps  in  Commission  oversight  are  attributable  in 
part  to  a  lack  of  technical  staff,  the  Chairman  should  also  determine  the 
number  of  staff  needed  to  adequately  oversee  the  rapid  growth  of  market 
automation  and  report  this  information  to  the  Commission’s  congressional 
appropriations  and  authorization  committees  in  time  for  consideration  in 
next  year’s  budget. 


Agency  Comments 
and  Our  Evaluation 


We  discussed  the  contents  of  this  report  with  senior  officials  from  nasd 
and  sec’s  Division  of  Market  Regulation.  We  incorporated  their  comments 
where  appropriate,  sec  officials  agreed  with  our  findings,  conclusions,  and 
recommendations.  Except  as  noted  below,  nasd  officials  also  agreed  with 
the  report. 
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NASD  officials  disagreed  with  our  characterization  that  their  software 
testing  approach  is  limited  or  incomplete  in  scope.  They  said  that  they 
have  adopted  a  rigorous  approach  to  testing.  In  addition,  NASD  said  that 
while  its  approach  may  differ  from  other  approaches,  it  is  successful,  as 
demonstrated  by  the  significant  number  of  changes  that  have  been 
introduced  over  the  years  without  problems.  Nevertheless,  nasd  officials 
told  us  that  as  an  act  of  caution,  they  will  engage  an  independent  reviewer 
to  assess  the  testing  function  and  will  respond  appropriately  to  the 
reviewer’s  recommendations.  We  believe  that  this  is  a  prudent  step. 


We  are  sending  copies  of  this  report  to  interested  congressional 
committees,  the  Chairman  of  the  Securities  and  Exchange  Commission, 
the  President  and  Chief  Executive  Officer  of  the  National  Association  of 
Securities  Dealers,  and  to  other  interested  parties.  Copies  wiU  also  be 
made  available  to  others  upon  request.  Please  call  me  at  (202)  512-6418  if 
you  or  your  staffs  have  questions  about  this  report.  Other  major 
contributors  are  hsted  in  appendix  I. 


Hazel  E.  Edwards 

Director,  Information  Resources  Management/ 
General  Government  Issues 
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Major  Contributors  to  This  Report 


Accounting  and 
Information 
Management  Division, 
Washington,  D.C. 


Linda  D.  Koontz,  Associate  Director 

Gary  N.  Mountjoy,  Senior  Evaluator 

William  D.  Hadesty,  Technical  Assistant  Director 

Kevin  G.  McCarthy,  Senior  Evaluator 

Sabine  R.  Paul,  Senior  Information  Systems  Analyst 


(511079) 
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